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® Method for correcting systems of coordinates In a robot having visual sensor device and apparatus therefor. 

@ The co-ordinates system correction apparatus, incorpo- 
rated in a robot system having a visual sensor for measuring 
two or three dimensional position and posture data, corrects 
a visual sensor co-ordinates system and e robot co-ordinates 
systems. The apparatus corrects the co-ordinates systems by 
means of a correction of parameters representing a relation- 
ship between the co-ordinates systems. The parameter 
correction is caused by calculating parameter errors from a 
deviation between calculated position and posture date and 
measured position and posture data. 
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METHOD FOR CORRECTING SYSTEMS OF COORDINATES 
IN A ROBOT HAVING VISUAL SENSOR DEVICE 
AND APPARATUS THEREFOR 
1 BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

The present invention relates to a control 
means for a robot having a visual sensor device for 
5 measuring a position and posture of an object in a two- 
or three-dimension, and more particularly to a co- 
ordinates system correction method and apparatus in which 
deviations between a visual co-ordinates system and a 
robot co-ordinates system are corrected whereby the 
10 operation of the robot with the visual sensor device may 
be carried out with high accuracy • 

DESCRIPTION OF THE PRIOR ART 

In general, environmental data from a visual 
sensor are defined by a proper co-ordinates system fixed 

15 to the visual sensor. This is because the position and 
posture of an object are detected as the position and 
posture of the object in the proper co-ordinates system 
fixed to the visual sensor. For instance, Japanese Patent 
Publication No. 36 374/75 discloses a method for measuring 

20 a position of an object by using a television camera in 
a triangular surveying manner. Also, in this case, the 
position of the object is known by using a co-ordinates 
system related to the television camera in accordance 
with the obtained video signals. 
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1 On the other hand, it i» convenient for carrying 

out various works that the robot works are defined by the 
movements of a tool mounted on a wrist of the robot. 
Therefore, it is desirable that the movement of the tool 

5 is defined in a proper co-ordinates system fixed to the 
wrist. 

Also, in the case where characteristics of the 
robot per se, such as, for example, a speed command value 
of the robot are desired to be calculated on the basis of 
10 the co-ordinates system proper to the robot body. 

Thus, it is desirable to use the co-ordinates 
systems proper to the wrist and the body of the robot 
and the visual sensor, respectively, in order to control 
the operations of the wrist, the body and the visual 
15 sensor. However, in the prior art, almost no attempt 

has been made to adopt the respective co-ordinate systems. 

Thus, in the conventional robot control system, 
the respective co-ordinates systems have not been used. 
Therefore, the most preferable co-ordinates system has 
20 not always been used for the respective structural 
components „ 

In this connection, in the conventional robot 
system having a visual sensor, specific data types 
corresponding to the respective visual sensor components 
25 have been used whereby the data are represented to 

variously drive the robot. Therefore, it is necessary 
to provide various data types for the various visual 
sensor components and prepare various soft wares for 
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1 driving the robot. Namely/ if, for example, the co-ordinates 
system for the visual sensor device is solely used, even 
if the same robot body and wrist are used, it is necessary 
to reform the overall data system for the co-ordinates 
5 system of the visual sensor device. Further, in the prior 
art, the specific data system for the various visual 
sensor units is used* As a result, upon controlling the 
robot, such a difficulty would be encountered that a 
command signal for the sensor feedback control in a stand- 

10 ardized manner with the robot language is given. 

Also, in the prior art, in the case where the 
robot is operated by use of the visual sensor, a planar 
binary or multivalue image is recognized by the visual 
sensor and the handling is carried out in accordance with 

15 such a recognition. Therefore, it is only possible to 
handle a planar object having a constant posture. At 
this time, it is sufficient to consider the parallel 
movement on a plane and the rotation in the plane as 
errors between the co-ordinates system of the robot and 

20 the co-ordinates system of the visual sensor as shown in 
USP 3,986,007, and it is relatively easy to correct the 
co-ordinates system. Such a correction is carried out 
in a recent planar visual sensor device which is available 
on the market. However, as the handling operation of the 

25 robot is increased and a object having complicated shape 
must be handled, it becomes necessary to recognize a 
three-dimensional object having a non-fixed position and 
posture as well as a planar object. In order to meet 
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1 this requirement, there has been proposed a t^Lf- 4 * 7 
dimensional recognition method using a dual visual sensor 
or distant image. In such a three-dimensional visual 
sensor, it is necessary to correct spacial errors be- 

5 tween the co-ordinates systems of the robot and of the 
visual sensor. However, such a technique has never been 
proposed or realized. 

SUMMARY OF THE INVENTION 

An object of the. present invention to provide 
10. a co-ordinates system correction method capable of 

effectively utilizing data from a visual sensor means by 
, using the respective co-ordinates systems for the respec- 
tive structural components and an apparatus for carrying 
out the same. 

15 Another object of the invention is to provide 

a co-ordinates system correction apparatus capable of 
readily coping with the position change of the visual 
sensor device and/or with the replacement of the visual 
sensor device. 

20 . . . Still another object of the invention is to 

provide a co-ordinates system correction method and 
apparatus capable of obtaining with ease and high accuracy 
a spacial relationship between the visual sensor co- 
ordinates system and at least one of the co-ordinates 

25 systems of the robot body and the robot hand. 

In order to attain the above-mentioned objects, 
according to the invention there is provided a co-ordinates 
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1 system correction apparatus in which a co-ordinates 
system is setted in at least one of robot body and a 
robot hand, the apparatus comprising visual sensor means 
to which a proper co-ordinates system is setted, means 
5 for detecting a position and/or posture of the object 

in the co-ordinate system in the visual sensor means, and 
parameter storing means for storing parameters determin- 
ing a relationship between the co-ordinates system of 
the visual sensor means and at least one of the co- 

10 ordinates systems of the robot body and the robot hand, 

wherein by using the co-ordinates transformation parameters, 
the position and/or posture of the object in the co- 
ordinates system of the visual sensor means are trans- 
formed into one in the co-ordinates system of the robot 

15 hand and/or the co-ordinates system of the robot body, 

or transformed into one in the co-ordinates system of the : 
robot body and further transformed into one in the co- 
ordinates system of. the robot body. 

According to the present invention, transformation 

20 parameters are established in the three co-ordinates 

systems of the visual sensor means, the robot hand, and 
the robot body (or in the two co-ordinates systems if the 
co-ordinates systems of the robot hand and the robot 
body are the same) , so that the environmental data in 

25 the proper co-ordinates system established in the visual 
sensor means may be used in other co-ordinates system. 
Therefore, it is advantageous that the respective co- 
ordinates systems may be used for the respective structural 
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1 components. In addition , the environmental data obtained 
from the visual sensor means may be effectively used. 

According to another aspect of the invention, 
a plural kinds of the co-ordinates trans formation para- 
5 meters are set and stored in the storing means whereby 
even if the visual sensor means is replaced by another 
visual one, the co-ordinates transformation parameters 
corresponding thereto may be assigned. In accordance 
with the specific embodiment, since plural kinds of 
10 transformation parameters are prepared, any parameter 

may be assigned corresponding to the various visual sensor 
means . Such a various correspondence to the various 
kinds of visual sensor means is actually available* It 
is necessary to fix the visual sensor means anywhere in 
15 the robot body* However, the visual sensor means is 
not limited to a single visual sensor means, and. it is 
general that the position where the visual sensor means 
is fixed is different in accordance with an object to 
be handled. Therefore, it is necessary to install the 
20 visual sensor means to a different position in accordance 
with the respective mounting positions and to use dif- 
ferent visual sensor means. It is therefore preferable 
to assign the corresponding parameters in accordance 
with the respective positions. As in the embodiment of 
25 the invention, if plural kinds of parameters are stored, 
when the visual sensor means is replaced by new one or 
changed in its position, it is possible to assigne the 
parameters corresponding to the changes. This is very 
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1 convenient* 

Also, it is necessary to measure the position 
and posture of the object and direct the robot hand to 
the posture direction in order to handle the object in 
5 any position and posture. To meet this requirement, it 
is necessary to accurately correct a deviation of the 
visual sensor co-ordinates system for measuring the posi- 
tion and posture of the object from the robot body or 
robot hand co-ordinates system in accordance with the 

10 parameters. By the way, even if the robot is moved in 
accordance with the position and posture information 
measured by the visual sensor device, the robot is moved 
to a position and posture deviated from the practical 
position and posture of the object. Therefore, by 

15 utilizing this feature inversely, in the robot body and/ 
or robot hand co-ordinates systems, the position and 
posture of the object known in advance are measured by 
the visual sensor means, and it is possible to obtain 
the parameters giving a relationship between the robot 

20 visual sensor and the robot body or hand co-ordinates 

systems from the errors between the measured values and 
the calculated values. In accordance with the present 
invention, by using the parameters in this manner, the 
correction between the robot body or hand co-ordinates 

25 system and the visual sensor co-ordinates system may be 
attained. 
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1 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig» 1 is a schematic view showing a robot 
system to which one embodiment of the invention is 
applied? 

5 Fig- 2 is a perspective view showing a co- 

ordinates system of a visual sensor shown in Fig., 1? 

Fig* 3 is a perspective view showing a co- 
ordinates system fixed to the object shown in Fig- 2? 

Fig. 4 is a schematic view illustrating a robot 
10 system to which a three-dimensional stationary visual 
sensor device is fixed? 

Figs. 5 and 6 are views showing a relationship 
between the robot co-ordinates system and the stationary 
visual sensor co-ordinates system; 
15 j Fig. 7 is a block diagram showing the functions 

of the: components of the system shown in Fig- 4? 

Fig- 8 is a view showing one robot system having 
a three-dimensional visual sensor? 

Fig- 9 is a view showing a relationship between 
20 the robot hand co-ordinates system and the robot visual 
sensor co-ordinates system? 

Fig- 10 is a view showing a relationship between 
the object and the robot hand with a visual sensor? and 

Fig. 11 is a block diagram showing the functions 
25 of the components of the system shown in Fig. 8- 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

A preferred embodiment of the invention will 
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1 now be described with reference to the accompanying 
drawings . 

In Fig, 1, there is shown a robot system provided 
with a co-ordinates system correction device in accordance 
5 with the preferred embodiment. Two visual devices 1 and 
2 are provided in this robot system. The stationary visual 
device 1 is secured to a base 5 on which a robot 3 is 
installed. The wrist visual device 2 is mounted on a 
wrist portion of the robot 3. The robot system shown has 

10 six degrees of freedom in motion. 

In this robot system, the visual devices 1 and 
2 may detect a position and a posture of an object 4. 
A co-ordinates system is set in a robot body. More 
specifically, stationary co-ordinates XYZ is fixed in a 

15 stationary portion between the robot 3 and the base 5. 
Another co-ordinates system is set in the wrist portion 
of the robot 3 and is,, in particular, composed of co- 
ordinates x^Y^ 2 ^ of a hand 6. On the other hand, proper 
co-ordinates systems are established for- the visual 

20 devices 1 and 2, respectively. Namely, a co-ordinates 
system x-jYj^ is fixed for the stationary visual device 
1 and a co-ordinates system * 2 y 2 z 2 is fixed for wrist 
visual device 2 m 

When the object 4 are detected on the basis 

25 of the co-ordinates systems x^y^^ and ^ 2 ^2 Z 2 by tile 

visual devices 1 and 2 through an image processor 8, the 
data from the visual devices 1 and 2 on the basis of the 
visual device co-ordinates systems are transformed into 
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1 the data on the basis of the co-ordinates systems of the 
robot 3 through parameters giving relationships between 
the co-ordinates systems of the visual devices 1 and 2, 
and the stationary co-ordinates XYZ and the co-ordinates 
5 a^, y h , z h of the hand 6. Thus, the data detected by 
the visual devices 1 and 2 on the basis of their own 
co-ordinates systems are transformed into the data on 
the basis of the co-ordinates systems of the robot body 
and of the wrist of the robot. The robot operates on the 
10 basis of such transformed data. In this manner, the 

structural components can desirably operate on the basis 
of the respective co-ordinates systems. The above- 
described parameters are stored in a parameter storing 
means incorporated in a robot control unit 7. 
15 : in this embodiment, a plural sets of parameters 

are provided and stored in the parameter storing means so 
as to readily cope with the position changes of the 
visual devices and/or the replacement of the visual 
devices . 

20 The construction and the operation of the 

embodiment will now be described in more detail. 

As described above, in this embodiment, a 
visual sensor system includes two visual devices 1 and 2. 
The stationary visual device 1 is fixed to the base 5 
25 and the wrist visual device 2 is mounted on the wrist 
portion of the robot 3. Each of the visual devices is 
composed of, e.g. a camera having a slit light source. 
.- The stationary visual device 1 is a two- or three- 
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dimensional visual sensor and can recognize the position 
and the posture of the object 4. For instance, if the 
stationary visual device 1 is a two-dimensional visual 
sensor, a position p of the object 4 in the co-ordinates 
system fixed to the visual device 1 as shown in Fig. 2 is 

defined by the components P x ^/ Pyl' Pzl of the X l' y l' 2 1 
axes and is expressed in the following equation (1) . 

■ P = Pxl h + p yl 3l + p zl *1 {1) 

where y^, and ic 1 are the unit vectors in parallel to 

the x 1# y 1 and z 1 axes, respectively. When the stationary 
visual device 1 is the two-dimensional sensor, in 
general, a direction of the z 1 axis is determined by the 
mountirig position of the visual sensor and the component 
p . is "a constant. On the other hand, the posture of 
the object 4 is expressed by cosine components of the, 
two axes fixed thereto and intersected at a right angle 
with each other as shown in Fig. 3. The cosine compo- 
nents are given by unit vectors f and g. When the object 
4 shown in Fig. 3 is viewed from above the axis ? with 
the two-dimensional visual sensor, the posture is given 
as follows . 



$ - jt x . (2) 

3 = g xl t l + 9yl ?1 (3) 
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If the stationary visual device 1 is of the 

three-dimensional type, it is possible to recognize any 
positions and postures of the object. In this case, the 
9 (nine) parameters of P xl ' P yl * ***' 9 y i' 3115 9 zl in 
the following eguations may be detected. 



P - Pxl *1 + p yi p zi S i 

1 - f xl *1 + £ yl 3 1 + f zl *1 
9 - 9 X 1 *1 + ^yl ^1 + g z i 



... (4) 



(5) 



(6) 



Since the stationary visual device 1 is fixed 
to the base 5, the relationship between to co-ordinates 
system>of the stationary visual device 1 and the stationary 
co-ordinates XYZ is unchangeable a fixed manner, and 
10 then the cosine components (P x / Py' Pz^ ' ^X' ^Y' ^Z* ' 
(g x . g y t S z > of P' ^' 9 in co-ordinates XYZ are 

given by the following relationship with using the data 
from the visual device 1. 



1 


0 




Px 


f x 




Py 


f y 


% 


Pz 


H 


*z 



^ 1 o 0 

Pxl f xl 9 xl 
Pyl f yl g yl 
P 2 1 f zl g zl 



(7) 



where A is a matrix of 4 x 4 and its all elements are 
15 constants . 
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1 In the same manner, the wrist visual device 2 

mounted on the wrist portion of the robot may detect the 
position and posture components (P x2 ' P y 2' p z2^ ' * f x2 f 

f y2' f z2 )f (g x2' *y2' 9 z2 ) of ^ ob 3 ect on **** basis 
5 of the characteristic co-ordinates system x 2 y 2 z 2 fixed 
in the visual device 2. Since the visual device 2 is 
fixed to the wrist of the robot, the co-ordinates system 
of the visual device 2 is in an unchangeable relation 
with the co-ordinates x h# y h , z h of the hand 6 also fixed 
10 to the wrist portion * The data of the position and 

posture of the object from the wrist visual device 2 are 
transformed into the data on the basis of the hand co- 
ordinates system through the following relationship. 




where B is a matrix of 4 x 4 and its all elements are 
15 constants. 

As described above, in this embodiment, the 
relationship between the co-ordinates system x 1# y^, z^ 
of the stationary visual device 1 and the stationary 
co-ordinates system X Y 2 is given by the parameters 
20 determined by the matrix A* Also, the relationship 

between the co-ordinates system x 2 # Y 2 ' z 2 of the wrist 
visual device 2 and the co-ordinates system x fa , y h , z h 



. 0 * # , • ♦ . 

1 of the hand 6 is given by the parameters determined by 
the matrix Bo Therefore, the data which the visual 
devices 1 and 2 recognize with their own co-ordinates 
systems, respectively, are transformed into the data on 

5 basis of the co-ordinates systems of the robot 3 and the 

hand 6 fixed to the wrist of the robot 3 through the above- 
described parameters „ The robot operates on such trans- 
formed data* These parameters are stored in the parameter 
storing means of the robot control unit 7 „ 
10 The embodied apparatus with such a construction 

will operate as follows with respect of the control of 
the robot „ 

: For instance, when it is assumed that the hand 

6 of the robot is controlled to be positioned in the 
15 same place as the position and the posture of the object 
are recognized by the visual sensors . In this case, 
it is preferable that the position and posture p, f , g 
of the hand 6 be given by the stationary co-ordinates 
system o Therefore, when the stationary visual device 1 
20 is used, the data from the visual device 1 are multiplied 
. . ._ by the matrix A of parameter and are transformed into 

the data on the stationary co-ordinates « This makes it 
possible to control the hand 6 by means of usage of such 
transformed data* When the wrist visual device 2 is 
25 used, in the same manner, the data from the visual device 
2 is used, in the same maimer, the data from the visual 
device 2 are multiplied by the matrix B of parameter 
and are transformed into the data on the hand co-ordinates 
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1 system. In this case, the transformed data may be further 

transformed into the data on the stationary co-ordinates 
system by using another parameters. These another 
parameters may be expressed by a matrix C, the matrix C 
5 iB of 4 x 4 type and its elements are functions of 6 (six) 
displacements of a kinematic pair 6^, Q^* e g/ i.e. 

c^ « f^ (6 lf 6 2 , , Sg) . From the values of p, f and 

g thus transformed into the stationary co-ordinates 
system, the displacements of the kinematic pair e lf / 9 2 f ' 

10 are calculated, each of which corresponds to 

the respective transformed values. Then, an actuator for 
driving the kinematic pair is controlled so that each 
dispacement of the respective pairing elements becomes 
the corresponding calculated value. 

15 , An explanation has been made as to the example^ 

of a robot system having two visual sensors. However, 
any number of the visual sensors may be used as far as 
parameters determining a relationship between the co- 
ordinates fixed to each visual sensor and the robot co- 

20 ordinates are defined. In this case, if the visual 

sensors .are numbered and then the number of a desired 
visual sensor is called out by a robot language or the 
like, the control unit 7 generates a drive signal to the 
desired visual sensor. Then, the data from visual 

25 sensors are multiplied by a matrix (parameters) stored 

in advance and corresponding to the called visual sensor, 
so that the data for controlling the robot may be obtained 
As described above, in the co-ordinates system 
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1 correction apparatus in accordance with this embodiment, 
since the visual devices 1 and 2 are provided with the 
respective co-ordinates system therefor and the 
relationships between the sensor co-ordinates systems 
5 and the robot co-ordinates systems are determined by 
the matrixes, the same drive soft ware may control the 
robot in accordance with each signal of the sensors even 
for the robot system having a plurality of visual sensors. 
Therefore, it is possible to standardize the drive soft 
10 ware. Also, if the matrices each corresponding to 

the respective sensors are stored in the control unit 7 
in advance, it is possible to carry out the feedback 
.control of the robot simply by calling the desired 
sensor number in terms of the robot language. A compli- 
15 cated robot language command may not be required. Further, 
if the parameters for data transformation are represented 
in a matrix form, it is possible to attain a high speed 
calculation through an array processor. If the matrix 
is simple, it is of course possible to make the data 
20 transformation through a particular sub-routine. 

In the preceding embodiments, although there 
is not shown a specific method for obtaining the trans- 
formation matrixes A or B in the case where the visual 
device 1 or 2 is of a three-dimensional visual sensor 
25 type, the co-ordinates system correction apparatus in 
accordance with the present invention may be used for 
that case. A method for obtaining the matrix A will 
now be described in accordance with another embodiment. 
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Pig. 4 is an illustration of one embodiment 
showing a parameter measuring method in relation to a 
three-dimensional visual sensor. Pig. 5 is a schematic 
view showing a relationship between the robot and the 
stationary visual sensor. Fig. 6 shows a relationship 
between the co-ordinates systems of the robot and the 
stationary visual sensor. Fig. 7 is a block diagram 
illustrating functions of components of the embodiment 
shown in Fig . 4 . 

In these figures, .a reference numeral 1 
designates a stationary visual device or sensor fixedly 
mounted onto a base 5 on which a robot 3 is installed. 
The robot 3 is provided at a distal end portion thereof 
with a hand 6. The stationary visual device 1 and the 
robot, 3 are associated with a robot control unit 8, 
an image processor 8, and a parameter calculator 9. 

It is assumed, as shown in Fig. 5 that the 0 
is an origin of the robot co-ordinates system comprising 
the axes of x Rf y R and z R , and the O v is an origin of 
the stationary visual sensor co-ordinates system com- 
prising the axes of x v , y v and z^, the relationship 
between the robot co-ordinates system and the stationary 
visual sensor co-ordinates system may be expressed by 
the following equations : 
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(10) 



1 where L = {L x , L^, I*^) is a vector representing the dis- 
placement from the origin O r of the robot co-ordinates 
system to the origin O v of the stationary visual sensor 
co-ordinates system, and M^, Mg, and are matrixes 

5 representing the three rotational displacements between 
the co-ordinate systems . These matrixes are expressed 
as follows, when angles of rotation a, &, and y are 
selected as shown in Fig. 6: 



10 0 
0 cos a sina 
0 -sina cosa 



Mo = 



cosB 0 
0 1 
sinB 0 



— sinB 
0 

cosB 



My 



cosY sinT 0 
-sinT cosT 0 
0 0 1 



(11) 



The robot control unit 7 commands the robot 3 to 

10 move to the position X = (x, y, z) and posture f « (f v * 

fy, ± z ) in terms of . the robot co-ordinates system. If 

the parameters L , L , L , a, and y are the same 

x y z 

values as the designed one and include no error r the 
robot must be moved to the following points: 
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X - x x R + y y R + z z R + $ R (12) 

1 = f x *R + f y h + £ z K •••• < 13 > 

1 To the contrary f in case that these position 

and the posture are calculated with using the data from 
the stationary visual sensor 1 and the calculated position 
and posture are expresseid as X 1 = (x f , y 1 , z 1 ), £• = 

5 «x" f y" f Z ' ) - 

X' - x f x v + y' y v + z» z v + 0 v (14) 

f ' = f , x +f,y + f , z . . . . (15) 

Z V y' J v Z V •••• UJ/ 

The relationships between x, y, z and x f , y 1 , z 1 , and 

between f^, f y , f g and f x f , f f , f z , are determined from 

the equations (9) to (11) . 

However, if the stationary visual sensor 1 is 

10 not accurately mounted on the predetermined portion of 

the robot system, the errors are expressed as AL of L , 

x x 

ALyr A L Z * Aa, AB, and Ay. In this case, the parameters 

in the equations (9) to (11) become (L + AL ) , (a + Aa) 

and the like. Therefore, even if the robot is commanded 

15 to move to X = (x, y, z) f f" «= (f , f , f ) , the robot 

x y z 

can not reside in the position and posture X' «= (x' r y', 
2'), ?' = (f x ,, f yI , f z .) but in X" = (x", y", z"), 

?" » (f x n ' f y "' f 2 n) which are deviated from X', r'. 
Therefore, if the respective deviations between x 1 , y'. 
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z' and x«, y", z«, and between f xf , f yf , f 2l and f x „, f H/ 
f zff are measured, AL x , AL y , AIj 2 , Aa, AB, Ay may be 
obtained from the equations (9) to (11) . 

The construction and operation of the embodiment 
shown in Fig, 4 will now be described with reference to 
Fig. 7. 

The robot 3 is moved in accordance with the 
position and posture command vectors X, ? from the robot 
control unit 7 (step 101) . The stationary visual sensor 
t 1 picks up an image of the robot 3 after the movement 
^thereof and feed the image information to the image 
processor 8 (step 102) . The position and posture X n , ? n 
of the robot 3 as viewed from the stationary visual 
sensor 1 are measured in the image processor 8 (step 103) . 
On the, other hand, the parameter calculator 9 calculates 
the position and posture X 1 , f 1 (equations (14) and (15)) 
in the stationary visual co-ordinates system by using 
the equations (9), (10) and (11) on the basis of the 
command signals or vectors X, 5 from the robot control 
unit 7 (step 104). If the stationary visual sensor 1 is 
accurately mounted on the predetermined portion of the 
robot system, the position X* » and posture f fl of the 
robot measured by the stationary sensor 1 coinside with 
the calculated position X 1 and posture f 1 of the robot. 
To the contrary, if the stationary sensor 1 is not 
accurately mounted on the predetermined portion of the 
robot system, the measured position X 11 and posture ? ,f 
are deviated from the calculated position X 1 and posture 
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1 ?' by some deviations caused by such unaccurate mounting 

of the robot. Namely, the measured position X' 1 and 
posture 1" have the deviations from the practical 
position and posture of the robot, which are caused by 
5 the parameter errors, e.g. AI* x . Accordingly, the correct 
parameters may be obtained from the respective deviations 
between X 1 and X", and between ?' and (steps 105, 
106). 

Also, in the above-described measurement of the 

10 parameters, even if the visual device is fixed to the 
wrist of the robot as the visual sensor 2, the matrix B 
may be obtained in the same manner. 

Pig. 8 is a schematic view illustrating one 
embodiment of a robot system having a three-dimensional 

15 movable, visual sensor, Fig. 9 shows the relationship 

between the robot hand co-ordinates system and the robot 
visual sensor co-ordinates. system, Pig. 10 shows the 
relationship between the object and the robot hand with 
the visual sensor more specifically, and Fig. 11 is a 

20 block diagram showing the functions of the components of 
the system shown in Fig. 8. 

In these figures, a reference numeral 2 
designates a wrist visual sensor mounted on a distal end 
portion of the robot for sensing an object 4 to. be. measured. 

25 The other elements and components are the same as shown 
in Fig. 4. 

As shown in Fig. 9, the parameters to be used 
upon mounting the robot wrist visual sensor 2 on the robot 



- 22 - 



0161417 



1 wrist include 6 (six) parameters such as a vector £ « (L x , 
L y' L z* representing a displacement between the origin 
O h of the robot hand co-ordinates system and the origin 
O v of the robot wrist visual sensor co-ordinates system, 

5 and angles of rotation a, B, y representing the respective 
rotational displacement between the corresponding axes 
of the co-ordinates systems. In this case, the relation- 
ships between the origin of the robot hand co-ordinates 
system and the origin 6^ of the wrist visual sensor co- 
10 _ordinates system, and between the axes x^, y fa , of 

the robot hand co-ordinates system and the axes x.^, y v , 
2 v of the wrist visual sensor co-ordinates system are 
represented by the following equations: 



v h x n y z h 



(16) 



x_ 



M M 0 M. 
T B a 







X h 




= M 








V 



, M = 



M Y M 6 M a 



(17) 



where M Q , and are the 3x3 matrixes representing the 
15 rotations about x h r y^ and z^, respectively, and are 



given as follows: 



10 0 
0 cosa sina 
0 -sina cosa 
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Mo = 
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cosB 0 -sinB 

0 10 
sinB 0 cosB 
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M__ = 



cosy siny 0 
-siny cosy 0 
0 0 1 



... (18) 



As shown in Fig. 10, if the normal vector fixed 

to the object 4 as viewed from the robot wrist visual 

sensor co-ordinates system is expressed as f = (f v , f v , 

f ) , the following equation is obtained: 
z 



f = f x x v 



+ f v y + f v t 

y ^v z v 



... (19) 



In order to express this vector in terms of the robot hand 
co-ordinates system, the vector is transformed into the 
following equation by expressing x v , y v , z v by x h , y h , z fa 
in accordance with the equations (17) and (18): 



f = f x, 
x h 



. (20) 



where the normal vector as viewed from the robot hand 

co-ordinates system is expressed by f « (f R , f R , f R ) . 

x y z 

Also, when a point on the object 4 is viewed 
from the visual sensor co-ordinate system, if the point 
is viewed as X = (x v , y v , z^) , the following equation is 
given: 



— 24 — 

X " x v x v + y v y v + z v 2 v •••• (21) 



1 If the scaling ratios of the television camera of the 

robot wrist visual sensor 2 are expressed by s x , s , 

s with respect to the respective directions x , v z 
^ v' -'v' v' 

X is viewed from the robot wrist co-ordinate system, 
5 and is expressed in accordance with the equation (16) 
as follows s 



X = L x X h + L y v h + L z z h + s x x v x v + 

s y y v y v + s v z v z v ..... ( 22) 



Y v / z v in this equation are represented by x" h , y fa , 
in accordance with the equations (17) and (18) and are 
substituted thereby. Then, the following equation is 
10 given s 



x = x h x h + y h y h + z h ^h ....(23) 

If the equation (22) is thus transformed, the fixed point 
as viewed from the robot wrist co-ordinate system is 
positioned as X = (x h , y fa , z h ) . 

As potential errors which would be generated 
15 when the robot wrist visual sensor 2 is mounted on the 
robot hand, errors Al> x , AL y , AL 2 , Act, AS, Ay would be 
considered with respect to L x , L yf L z , a, 0, Y , respectively. 
When these errors would be present, the parameters in the 
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1 equations (16) to (18) such as L x are replaced by (L x + 

AL x ) and the like. In this case, the equations (16) and 



(17) may be rewritten as follows: 







-*■ 






o 






V 
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-4- 








= G 
















Z 






V 






\ / 







g ll g 12 ^13 9 14 

q = g 21 g 22 ^23 9 23 

g 31 g 32 9 33 g 34 

g 4l 9 42 g 43 g 44 



(24) 

The element g ±j of the matrix G includes errors AL and 
5 the like. If is known, AL x , AL y , AL g , Act, A3, AY 

may be known by conducting suitable calculations. 

Unless errors exist, the object 4 which resides 
in a position X = ( Xy , y v , and in a posture ? = 

/-V _V ' _V. 

* r x' y' f z ) as viewed through the robot wrist visual 
10 sensor 2 may reside in a position X = (x h , y h , z h > and 
a posture f = <f*, f R , f R, as vieW ed through the robot 
hand co-ordinates. However, actually, due to the errors 
such as AL x , it is measured that the object 4 resides in 

a position X' = (x h , y£, 2 £) and in a posture f' = .(■f R, r 

R' R ' x 
f y / f z ) . By these deviations of the position and 

posture, the equations (16) and (17) are deformed into 
the equation (24) . Accordingly, by measuring the 
deviations in position and posture, it is possible to 
determine the elements of the matrix G. If the 

20 elements g ±j are known, the errors such as AL may be 
obtained. 



15 
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1 The construction and operation of the specific 

embodiment will now be described with reference to Pigs. 
8 and 11. 

The robot 3 is moved close to the fixed point 
5 on the object 4 in accordance with the command signals 

- of the position and posture from the robot control unit 

- 7 (step 201) . Thereafter, the image information is 
inputted into the robot wrist visual sensor 2 and fed to 
the image processor 8 (step 202). The position and 

10 posture X 1 and ?' are measured in the image processor 8 
- (step 203) . 

: On the other hand, the position and posture of 

the fixed point on the object is inputted in advance in 
the parameter calculator 9 (step 204) . On the basis of 
15 the yrist position and posture information (step 205) 

froid the robot control unit 7 , the position and posture 
X and ? of the fixed point as viewed from the robot wrist 
visual sensor co-ordinates system are calculated in 
accordance with the equations (16) , (17) and (18) (step 
20 206). Upon the calculation of X, it is made on the 
: basis of that there is no error in the parameters whereas 
the X' , J 1 measured by the robot wrist visual sensor 2 
include the parameter errors. Therefore f the vectors X 
and X 1 and 1 and l f are different from each other. Then, 
25 if the matrix G in the equation (24) is obtained in 

accordance with X 1 , £' (step 207), the parameters included 
in the respective components of the matrix G include 
errors. Accordingly/ by comparing the parameters in the 
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1 equations (16) and (17) to the parameters in the matrix 
G, it is possible to obtain the parameter errors (step 
208) . 

As described above, in accordance with the 
5 present invention, in a co-ordinate system correction 
apparatus including visual sensor means for detecting 
a position and/or posture of an object, the apparatus 
comprises means for setting a co-ordinate system in at 
least one of a robot body and a robot wrist, setting a 

10 proper co-ordinate system -in the visual sensor means and 
detecting the position and/or posture of the object in the 
co-ordinate system in the visual sensor means, and para- 
meter storing means for storing parameters determining a 
relationship between the co-ordinate system of the 

15 visual sensor means and at least one of the co-ordinate 
systems of the robot body and the robot wrist. By using 
the co-ordinate transformation parameters, the position 
and/or posture of the object in the co-ordinate system 
of the visual sensor means are transformed into the 

20 co-ordinate system of the robot wrist and the co-ordinate 
system of the robot body, or otherwise into the co-ordinate 
system of the robot body and further transformed into the 
co-ordinate system of the robot body. Accordingly, the 
respective structural elements such as the visual sensor 

25 means are operated by the respective co-ordinate systems. 
Thus, by the operation in a desired condition, it is 
advantageous that the obtained data are effectively 
utilized. Also, it is easy to carry out the co-ordinate 
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1 transformation with the parameters . By storing the 

parameters in the parameter storing means, it is possible 
to always correspond to the transformation of the data 
from the visual sensor means. 
5 Furthermore, according to the invention, by 

preparing and using a plurality of parameters, it is 
possible to correspond to the case where the visual 
sensor is changed in position and another visual sensor 
is used. 

!0 Also, it is possible to construct the robot 

system so as to command the robot in a standardized manner 
with a simple robot language. 

Moreover, in accordance with the present 
invention, since three-dimensional mounting parameters 

15 generated when the visual sensor is fixed to the robot 
co-ordinates system may be obtained by moving the robot, 
the following meritorious advantages can be obtained. 

(1) By obtaining the three-dimensional mounting 
parameters and using them, it is possible to handle with 

20 high accuracy a three-dimensional object taking any 

position and posture as well as a planar object by means 
of the visual sensors. 

(2) Also, the mounting parameters may be 
obtained simply by measuring the object while moving the 

25 robot. Accordingly, the correction work can be made 

more readily than the measurement of the mounting para- 
meters with a measuring tools. This leads to enhancement 
in accuracy of the robot operation and in efficiency. 
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1 It is apparent that the present invent f n 1 V ? 

not limited to the embodiments shown but various changes 
or modification are possible for those skilled in the 
art . 
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CLAIMS 

l w a robot system having a visual sensor comprisings 

a robot having a first co-ordinates system 
provided in at least one of a robot body and a robot 
hand; 

a visual sensor device having a second co- 
ordinates system for detecting a position and/or posture 
of an object? 

means for measuring parameters determining a 
relationship between said second co-ordinates system and 
said first co-ordinates system? 

means for storing said parameters measured by 
means of said measuring means; and 

means for transforming a detected position and/or 
posture information of the object in said second co- 
ordinatfes system into 1 an information in said first co- 
ordinates system through said measured parameters <, 
2 0 A robot system as claimed in Claim 1, wherein 

said parameters for co-ordinates transformation includes 
a plurality of sets of parameters each set corresponding to 
a respective visual sensor devices whereby a corresponding 
co-ordinates transformation parameter set may be assigned 
even if the visual sensor device is replaced by another 
visual sensor device . 

3 0 a robot system as claimed in Claim 2 r wherein 

said co-ordinates transformation parameter sets include 
one for a visual sensor device mounted on said robot 
hand and one for a visual sensor device mounted fixedly 
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on a base on which said robot is installed. 01 61417 
4 - A robot system as claimed in Claim 1, wherein 

said co-ordinates transformation parameters are deter- 
mined by a matrix. 

5 * A robot system as claimed in Claim 1, wherein 

said first co-ordinates system is established in said 
robot hand, and after an information in said second 
co-ordinates system is transformed into an information 
in said first co-ordinates system, said transformed 
information is further transformed into an information 
in a co-ordinates system of said robot body. 
6 - A robot system as claimed in Claim 1, wherein 

said parameter measuring means comprises: 

first means for moving said robot to a fixed 
point on the fixed object to be measured with said robot 
bein^f kept in a constant position and posture, and for 
calculating a position and posture of said robot hand -■■ 
moved said fixed point, 

second means for measuring a position and posture 
of said fixed point with a visual sensor device, and 

third means for calculating co-ordinates 
trans forma tipn parameters by means of data obtained by 
said first means and of data obtained by said second 
means . 

A robot system as claimed in Claim 6, wherein 
said visual sensor device includes a wrist visual sensor 
device mounted on said robot hand, 

8 - A robot system as claimed in Claim 1, wherein 
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said parameter measuring means comprises! 

first means for moving said robot to a commanded 
position and for calculating a position and posture in 
said commanded position, 

second means for measuring a position and 
posture of said robot after movement through said visual- 
sensor device, and 

third means for calculating co-ordinates 
transformation parameters by means of said calculation 
fesult obtained by said first- means and of said measure- 
ment result obtained by said second means. 
9 o A robot system as claimed in Claim 8, wherein 

said visual sensor device includes a stationary visual 
sensor device fixed to a base on which said robot is 
installed. 

10. A co-ordinates system correction apparatus 

comprising: 

a robot having a first co-ordinates system 

provided in at least one of a robot body and a robot 
hand; 

a visual sensor device mounted on a distal end 
portion of said robot or a base on which said robot 
is installed, said visual sensor device having a second 
co-ordinates system and a function for measuring a posi- 
tion and/or a posture of an object; 

visual data storing means for storing a posi- 
tion and/or a posture information measured by said visual 
sensor device. 
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robot data storing means for storing data 

representing a position and/or a posture of said robot 
in said first co-ordinates system when said visual data 
are measured; 

parameter measuring means for measuring para- 
meters giving a relationship between said first co- 
ordinates system and said second co-ordinates system by 
means of said visual data storing means and said robot 
data storing means; 

parameter storing means for storing said measured 
parameters; and 

means for transforming an, information of a 
position and/or a posture of an object in said second co- 
ordinates system into an information in said first co- 
ordinates system by using said parameters. 
11 • A method for correcting a robot visual sensor 

co-ordinates system for a robot system including a robot, 
a stationary visual sensor device fixed close to said 
robot , an image processor for processing image data from 
said sensor device , an error calculating device and a 
robot control unit for controlling and processing said 
components and data, said method comprising the steps of: 

moving said robot to a commanded position 
under a control of said robot control unit, 

measuring a position and posture of said robot 
in a stationary visual sensor co-ordinates system 
by means of said stationary visual sensor device and 
said image processor after movement of said robot, 
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calculating a commanded position and posture of 
said robot in said stationary visual sensor co-ordinates 
system by means of said error calculating device, 

calculating errors in mounting said stationary 
visual sensor device from a deviation between said 
calculated and said measured position and posture of said 
rrobot, and 

correcting the deviation between a robot co- 
ordinates system and said stationary visual sensor co- 
ordinates system, which caused by said mounting errors. 
12 • A method for correcting a robot wrist visual 

sensor co-ordinates system for a robot system including a 
robot, a sensor device mounted on a robot wrist, an image 
processor for processing image data from said wrist 
sensor \ device, a mounting error measuring device and a 
robot control unit for controlling and processing said 
components and data, said method comprising the steps 
of: 

moving said robot to a commanded point close to 
a fixed point of a stationary object to be measured - 
fixed in a constant position and posture under a control 
of said robot control unit, 

calculating a position and posture of said 
robot hand, 

measuring a position and posture of said fixed 
point in a robot wrist visual sensor co-ordinates system 



BNSDOCID: <EP 015U17A1_I_> 



- 35 - 01*1*17 

by means of said robot wrist visual sensor device and said 
image processor, 

- -calculating mounting errors of said robot wrist 
visual sensor from said measured and said calculated 
position and posture of said robot wrist by means of said 
mounting error calculating device, and 

correcting a deviation between, said robot wrist 
co-ordinates system and said robot wrist visual sensor 
co-ordinates system* 
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FIG. I 
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